The present invention relates to a speech synthesizer based on concatenation of digitally sampled speech units from a large database of such samples and associated phonetic, symbolic, and numeric descriptors.
A concatenation-based speech synthesizer uses pieces of natural speech as building blocks to reconstitute an arbitrary utterance. A database of speech units may hold speech samples taken from an inventory of pre-recorded natural speech data. Using recordings of real speech preserves some of the inherent characteristics of a real person""s voice. Given a correct pronunciation, speech units can then be concatenated to form arbitrary words and sentences. An advantage of speech unit concatenation is that it is easy to produce realistic coarticulation effects, if suitable speech units are chosen. It is also appealing in terms of its simplicity, in that all knowledge concerning the synthetic message is inherent to the speech units to be concatenated. Thus, little attention needs to be paid to the modeling of articulatory movements. However speech unit concatenation has previously been limited in usefulness to the relatively restricted task of neutral spoken text with little, if any, variations in inflection.
A tailored corpus is a well-known approach to the design of a speech unit database in which a speech unit inventory is carefully designed before making the database recordings. The raw speech database then consists of carriers for the needed speech units. This approach is well-suited for a relatively small footprint speech synthesis system. The main goal is phonetic coverage of a target language, including a reasonable amount of coarticulation effects. No prosodic variation is provided by the database, and the system instead uses prosody manipulation techniques to fit the database speech units into a desired utterance.
For the construction of a tailored corpus, various different speech units have been used (see, for example, Klatt, D. H., xe2x80x9cReview of text-to-speech conversion for English,xe2x80x9d J. Acoust. Soc. Am. 82(3), September 1987). Initially, researchers preferred to use phonemes because only a small number of units was required-approximately forty for American Englishxe2x80x94keeping storage requirements to a minimum. However, this approach requires a great deal of attention to coarticulation effects at the boundaries between phonemes. Consequently, synthesis using phonemes requires the formulation of complex coarticulation rules.
Coarticulation problems can be minimized by choosing an alternative unit. One popular unit is the diphone, which consists of the transition from the center of one phoneme to the center of the following one. This model helps to capture transitional information between phonemes. A complete set of diphones would number approximately 1600, since there are approximately (40)2 possible combinations of phoneme pairs. Diphone speech synthesis thus requires only a moderate amount of storage. One disadvantage of diphones is that they lead to a large number of concatenation points (one per phoneme), so that heavy reliance is placed upon an efficient smoothing algorithm, preferably in combination with a diphone boundary optimization. Traditional diphone synthesizers, such as the TTS-3000 of Lernout and Hauspie Speech And Language Products N. V., use only one candidate speech unit per diphone. Due to the limited prosodic variability, pitch and duration manipulation techniques are needed to synthesize speech messages. In addition, diphones synthesis does not always result in good output speech quality.
Syllables have the advantage that most coarticulation occurs within syllable boundaries. Thus, concatenation of syllables generally results in good quality speech. One disadvantage is the high number of syllables in a given language, requiring significant storage space. In order to minimize storage requirements while accounting for syllables, demi-syllables were introduced. These half-syllables, are obtained by splitting syllables at their vocalic nucleus. However the syllable or demi-syllable method does not guarantee easy concatenation at unit boundaries because concatenation in a voiced speech unit is always more difficult that concatenation in unvoiced speech units such as fricatives.
The demi-syllable paradigm claims that coarticulation is minimized at syllable boundaries and only simple concatenation rules are necessary. However this is not always true. The problem of coarticulation can be greatly reduced by using word-sized units, recorded in isolation with a neutral intonation. The words are then concatenated to form sentences. With this technique, it is important that the pitch and stress patterns of each word can be altered in order to give a natural sounding sentence. Word concatenation has been successfully employed in a linear predictive coding system.
Some researchers have used a mixed inventory of speech units in order to increase speech quality, e.g., using syllables, demi-syllables, diphones and suffixes (see, Hess, W. J., xe2x80x9cSpeech Synthesisxe2x80x94A Solved Problem, Signal processing VI: Theories and Applications,xe2x80x9d J. Vandewalle, R. Boite, M. Moonen, A. Oosterlinck (eds.), Elsevier Science Publishers B. V., 1992).
To speed up the development of speech unit databases for concatenation synthesis, automatic synthesis unit generation systems have been developed (see, Nakajima, S., xe2x80x9cAutomatic synthesis unit generation for English speech synthesis based on multi-layered context oriented clustering,xe2x80x9d Speech Communication 14 pp. 313-324, Elsevier Science Publishers B. V., 1994). Here the speech unit inventory is automatically derived from an analysis of an annotated database of speechxe2x80x94i.e. the system xe2x80x98learnsxe2x80x99 a unit set by analyzing the database. One aspect of the implementation of such systems involves the definition of phonetic and prosodic matching functions.
A new approach to concatenation-based speech synthesis was triggered by the increase in memory and processing power of computing devices. Instead of limiting the speech unit databases to a carefully chosen set of units, it became possible to use large databases of continuous speech, use non-uniform speech units, and perform the unit selection at run-time. This type of synthesis is now generally known as corpus-based concatenative speech synthesis.
The first speech synthesizer of this kind was presented in Sagisaka, Y., xe2x80x9cSpeech synthesis by rule using an optimal selection of non-uniform synthesis units,xe2x80x9d ICASSP-88 New York vol.1 pp. 679-682, IEEE, April 1988. It uses a speech database and a dictionary of candidate unit templates, i.e. an inventory of all phoneme sub-strings that exist in the database. This concatenation-based a synthesizer operates as follows.
(1) For an arbitrary input phoneme string, all phoneme sub-strings in a breath group are listed,
(2) All candidate phoneme sub-strings found in the synthesis unit entry dictionary are collected,
(3) Candidate phoneme sub-strings that show a high contextual similarity with the corresponding portion in the input string are retained,
(4) The most preferable synthesis unit sequence is selected mainly by evaluating the continuities (based only on the phoneme string) between unit templates,
(5) The selected synthesis units are extracted from linear predictive coding (LPC) speech samples in the database,
(6) After being lengthened or shortened according to the segmental duration calculated by the prosody control module, they are concatenated together.
Step (3) is based on an appropriateness measurexe2x80x94taking into account four factors: conservation of consonant-vowel transitions, conservation of vocalic sound s succession, long unit preference, overlap between selected units. The system was developed for Japanese, the speech database consisted of 5240 commonly used words.
A synthesizer that builds further on this principle is described in Hauptmann, A. G., xe2x80x9cSpeakEZ: A first experiment in concatenation synthesis from a large corpus,xe2x80x9d Proc. Eurospeech ""93, Berlin, pp.1701-1704, 1993. The premise of this system is that if enough speech is recorded and catalogued in a database, then the synthesis consists merely of selecting the appropriate elements of the recorded speech and pasting them together. It uses a database of 115,000 phonemes in a phonetically balanced corpus of over 3200 sentences. The annotation of the database is more refined than was the case in the Sagisaka system: apart from phoneme identity there is an annotation of phoneme class, source utterance, stress markers, phoneme boundary, identity of left and right context phonemes, position of the phoneme within the syllable, position of the phoneme within the word, position of the phoneme within the utterance, pitch peak locations.
Speech unit selection in the SpeakEZ is performed by searching the database for phonemes that appear in the same context as the target phoneme string. A penalty for the context match is computed as the difference between the immediately adjacent phonemes surrounding the target phoneme with the corresponding phonemes adjacent to the database phoneme candidate. The context match is also influenced by the distance of the phoneme to its left and right syllable boundary, left and right word boundary, and to the left and right utterance boundary.
Speech unit waveforms in the SpeakEZ are concatenated in the time domain, using pitch synchronous overlap-add (PSOLA) smoothing between adjacent phonemes. Rather than modify existing prosody according to ideal target values, the system uses the exact duration, intonation and articulation of the database phoneme without modifications. The lack of proper prosodic target information is considered to be the most glaring shortcoming of this system.
Another approach to corpus-based concatenation speech synthesis is described in Black, A. W., Campbell, N., xe2x80x9cOptimizing selection of units from speech databases for concatenative synthesis,xe2x80x9d Proc. Eurospeech ""95, Madrid, pp. 581-584, 1995, and in Hunt, A. J., Black, A. W., xe2x80x9cUnit selection in a concatenative speech synthesis system using a large speech database,xe2x80x9d ICASSP-96, pp. 373-376, 1996. The annotation of the speech database is taken a step further to incorporate acoustic features: pitch (F0), power and spectral parameters are included. The speech database is segmented in phone-sized units. The unit selection algorithm operates as follows:
(1) A unit distortion measure Du(ui, ti) is defined as the distance between a selected unit ui and a target speech unit ti, i.e. the difference between the selected unit feature vector {uf1, uf2, . . . ufn} and the target speech unit vector {tf1, tf2, . . . , tfn} multiplied by a weights vector Wu{w1, w2, . . . , wn}.
(2) A continuity distortion measure Dc(ui, uixe2x88x921) is defined as the distance between a selected unit and its immediately adjoining previous selected unit, defined as the difference between a selected units unit""s feature vector and its previous one multiplied by a weight vector Wc.
(3) The best unit sequence is defined as the path of units from the database which minimizes:       ∑          i      =      1        n    ⁢      xe2x80x83    ⁢            (                                                  D              c                        ⁡                          (                                                u                  i                                ,                                  u                                      i                    -                    1                                                              )                                *                      W            c                          +                                            D              u                        ⁡                          (                                                u                  i                                ,                                  t                  i                                            )                                *                      W            u                              )        .  
where n is the number of speech units in the target utterance.
In continuity distortion, three features are used: phonetic context, prosodic context, and acoustic join cost. Phonetic and prosodic context distances are calculated between selected units and the context (database) units of other selected units. The acoustic join cost is calculated between two successive selected units. The acoustic join cost is based on a quantization of the mel-cepstrum, calculated at the best joining point around the labeled boundary.
A Viterbi search is used to find the path with the minimum cost as expressed in (3). An exhaustive search is avoided by pruning the candidate lists at several stages in the selection process. Units are concatenated without doing any signal processing (i.e., raw concatenation).
A clustering technique is presented in Black, A. W., Taylor, P., xe2x80x9cAutomatically clustering similar units for unit selection in speech synthesis,xe2x80x9d Proc. Eurospeech ""97, Rhodes, pp. 601-604, 1997, that creates a CART (classification and regression tree) for the units in the database. The CART is used to limit the search domain of candidate units, and the unit distortion cost is the distance between the candidate unit and its cluster center.
As an alternative to the mel-cepstrum, Ding, W., Campbell, N., xe2x80x9cOptimising unit selection with voice source and formants in the CHATR speech synthesis system,xe2x80x9d Proc. Eurospeech ""97, Rhodes, pp. 537-540,1997, presents the use of voice source parameters and formant information as acoustic features for unit selection.
Each of the references mentioned above is hereby incorporated herein by reference.
In one embodiment, the invention provides a speech synthesizer. The synthesizer of this embodiment includes:
a. a large speech database referencing speech waveforms, wherein the database is accessed by polyphone designators;
b. a speech waveform selector, in communication with the speech database, that selects waveforms referenced by the database using polyphone designators that correspond to a phonetic transcription input; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In a further related embodiment, the polyphone designators are diphone designators. Optionally, the speech waveform selector uses criteria that (i) favor waveform candidates based, at least in part, directly on high-level linguistic features, and (ii) favor approximately equally all waveform candidates in respect to low-level prosody features except those wherein the low-level prosody features are unlikely. In a related set of embodiments, the synthesizer also includes (i) a digital storage medium in which the speech waveforms are stored in speech-encoded form; and (ii) a decoder that decodes the encoded speech waveforms when accessed by the waveform selector.
Also optionally, the synthesizer operates to select among waveform candidates without recourse to specific target duration values or specific target pitch contour values over time. In further related embodiments, the criteria include a first requirement favoring waveform candidates having pitch within a range determined as a function of high-level linguistic features. The criteria may also include a second requirement favoring waveform candidates having a duration within a range determined as a function of high-level linguistic features. Furthermore, the criteria may include a third requirement favoring waveform candidates having coarse pitch continuity within a range determined as a function of high-level linguistic features. Optionally, the criteria may be implemented by cost functions, and the requirement is implemented using a function having steep sides and a region that approximates a flat bottom.
In another embodiment, there is provided a speech synthesizer using a context-dependent cost function, and the embodiment includes:
a. a large speech database;
b. a target generator for generating a sequence of target feature vectors responsive to a phonetic transcription input;
c. a waveform selector that selects a sequence of waveforms referenced by the database, each waveform in the sequence corresponding to a first non-null set of target feature vectors,
wherein the waveform selector attributes, to at least one waveform candidate, a node cost, wherein the node cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost is determined using a cost function that varies nontrivially according to a second non-null set of target feature vectors in the sequence; and
d. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In a further related embodiment, the first and second sets are identical. Alternatively, the second set is proximate to the first set in the sequence. In another related embodiment, the second set is a function of the first set.
In another embodiment, there is provided a speech synthesizer with a context-dependent cost function, and the embodiment includes:
a. a large speech database;
b. a target generator for generating a sequence of target feature vectors responsive to a phonetic transcription input;
c. a waveform selector that selects a sequence of waveforms referenced by the database,
wherein the waveform selector attributes, to at least one ordered sequence of two or more waveform candidates, a transition cost, wherein the transition cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost is determined using a cost function that varies nontrivially according to the features of a region in the phonetic transcription input; and
d. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output. In another embodiment, a speech synthesizer includes:
a. a large speech database;
b. a waveform selector that selects a sequence of waveforms referenced by the database,
wherein the waveform selector attributes, to at least one waveform candidate, a cost, wherein the cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost is determined using a cost function that has at least one steep side; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In a further related embodiment, the cost function has a plurality of steep sides.
Another embodiment of the present invention provides a speech synthesizer, and the embodiment includes:
a. a large speech database;
b. a waveform selector that selects a sequence of waveforms referenced by the database, wherein the waveform selector attributes, to at least one waveform candidate, a cost,
wherein the cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost is determined using a cost function that has a region that approximates a flat bottom; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In further embodiments, the individual cost function is piecewise linear. Alternatively or in addition, the individual cost function is asymmetric.
In a further embodiment, there is provided a speech synthesizer, and the embodiment provides:
a. a large speech database;
b. a waveform selector that selects a sequence of waveforms referenced by the database,
wherein the waveform selector attributes, to at least one waveform candidate, a cost, wherein the cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost of a symbolic feature is determined using a non-binary numeric function; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In a related embodiment, the symbolic feature is one of the following: (i) prominence, (ii) stress, (iii) syllable position in the phrase, (iv) sentence type, and (v) boundary type. Alternatively or in addition, the non-binary numeric function is determined by recourse to a table. Alternatively, the non-binary numeric function may be determined by recourse to a set of rules.
In yet another embodiment, there is provided a speech synthesizer, and the embodiment, includes:
a. a large speech database;
b. a target generator for generating a sequence of target feature vectors responsive to a phonetic transcription input;
c. a waveform selector that selects a sequence of waveforms referenced by the database, each waveform in the sequence corresponding to a first non-null set of target feature vectors,
wherein the waveform selector attributes, to at least one waveform candidate, a cost, wherein the cost is a function of weighted individual costs associated with each of a plurality of features, and wherein the weight associated with at least one of the individual costs varies nontrivially according to a second non-null set of target feature vectors in the sequence; and
d. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In further embodiments, the first and second sets are identical. Alternatively, the second set is proximate to the first set in the sequence. In a related embodiment, the second set is a function of the first set.
In another embodiment, there is provided a speech synthesizer, and the embodiment includes:
a. a large speech database;
b. a waveform selector that selects a sequence of waveforms referenced by the database,
wherein the waveform selector attributes, to at least one waveform candidate, a waveform cost, wherein the waveform cost is a function of individual costs associated with each of a plurality of features, and wherein calculation of the waveform cost is aborted after it is determined that the waveform cost will exceed a threshold; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In another embodiment, there is provided a speech synthesizer, and the embodiment includes:
a. a large speech database referencing speech waveforms, wherein the database is accessed by polyphone designators;
b. a waveform selector that selects a sequence of waveforms referenced by the database,
wherein the waveform selector attributes, to at least one ordered sequence of two or more waveform candidates, a transition cost, wherein the transition cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost is determined by using, as an argument, an acoustic distance value selected from one of a first set of tables, each table in the first set corresponding to a non-null set of phonemes; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
In further embodiments, the acoustic distance is spectral distance and each table in the first set corresponds to a different phoneme. Optionally, the first set of tables is the result of vector quantization of spectra.
In another embodiment, there is provided a speech synthesizer, and the embodiment includes:
a. a large speech database referencing speech waveforms, wherein the database is accessed by polyphone designators;
b. a waveform selector that selects a sequence of waveforms referenced by the database,
wherein the waveform selector attributes, to at least one ordered sequence of two or more waveform candidates, a transition cost, wherein the transition cost is a function of individual costs associated with each of a plurality of features, and wherein at least one individual cost is determined by using as an argument for its function a phoneme-dependent acoustic distance measure; and
c. a speech waveform concatenator in communication with the speech database that concatenates the waveforms selected by the speech waveform selector to produce a speech signal output.
Another embodiment provides a speech synthesizer, and the embodiment includes:
a. a speech database referencing speech waveforms;
b. a speech waveform selector, in communication with the speech database, that selects waveforms referenced by the database using designators that correspond to a phonetic transcription input; and
c. a speech waveform concatenator, in communication with the speech is database, that concatenates waveforms selected by the speech waveform selector to produce a speech signal output,
wherein, for at least one ordered sequence of a first waveform and a second waveform, the concatenator selects (i) a location of a trailing edge of the first waveform and (ii) a location of a leading edge of the second waveform, each location being selected so as to produce an optimization of a phase match between the first and second waveforms in regions near the locations.
In related embodiments, the phase match is achieved by changing the location only of the leading edge and by changing the location only of the trailing edge. Optionally, or in addition, the optimization is determined on the basis of similarity in shape of the first and second waveforms in the regions near the locations. In further embodiments, similarity is determined using a cross-correlation technique, which optionally is normalized cross correlation. Optionally or in addition, the optimization is determined using at least one non-rectangular window. Also optionally or in addition, the optimization is determined in a plurality of successive stages in which time resolution associated with the first and second waveforms is made successively finer. In a further embodiment, the time resolution associated with the first and second waveforms in an initial one of the stages is downsampled by a factor that is a power of 2.